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1. Introduction 

Hopfield-type models pQ are recurrent networks of binary neurons (or Ising spins) 
with specific types of pair-interactions, designed to store and retrieve information in a 
distributed way. The main interest in their properties relates to the regime of operation 
close to saturation, where the number of patterns stored p scales with the number of 
bonds per neuron c (i.e. a = p/c remains finite as iV — ► oo). Over the years they 
have been studied intensively using statistical mechanical tools, in various connectivity 
versions. Rigorous analysis, based on techniques borrowed from spin-glass theory, 
has been carried out for Ising spin neural networks with full connectivity [3J EJ El El 
(c,N — > oo, c/N = 1), and for asymmetric |7J |K| and symmetric extreme dilution jU] 
(c, iV — > oo, c/N — ► 0). The approach of E] also proved effective for solving models 
with continuous neurons and discrete time dynamics JU]- More recently, attention has 
turned to finite connectivity attractor neural networks ^T] (c = 0(1), iV — > oo). 

In this paper we study the dynamics of asymmetrically extremely dilute graded 
response attractor neural networks with Langevin dynamics, near saturation. Such 
systems violate detailed balance in two ways, firstly due to the interaction asymmetry 
and secondly by the presence of a non-linear gain function (required for firing rates to 
saturate). Equilibrium statistical mechanics no longer applies; the system will never be 
in equilibrium. Yet, one still expects evolution to a stationary state, albeit one with 
microscopic probability currents. The stationary state of fully connected networks of 
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graded response neurons with continuous time dynamics [2] close to saturation have 
been studied at T = 0, ^21 El E] together with dynamics at arbitrary T but away 
from saturation ^Hj- In the present case of asymmetric extreme dilution one might 
expect the ideas of [7J |H] to apply, and anticipate the familiar simplification resulting 
from Gaussian distributed fields. This turns out to be wrong. In (3 |H] and (THj the spins 
were (stochastically) aligned to local fields in parallel, at discrete intervals. Spins at time 
t+1 did not depend on spins at time t other than via the fields. Due to the asymmetric 
dilution, the field distribution was Gaussian on finite time-scales (characterized by two 
scalars), leading directly to simple iterative laws for one or two scalar order parameters. 
In contrast, although here the fields still have a Gaussian distribution, the Langevin 
dynamics makes our model qualitatively different: spin updates now depend strongly 
on present values, in addition to the fields, and we will therefore need the (non- Gaussian) 
joint spin-field distribution. 

In this paper we use the generating functional analysis method of JH] > which has a 
strong record in the area of asymmetric disordered spin systems (e.g. [T7 | ITH t IT? H I20J). 
following closely the exposition in !15 i which allows us to be compact. As always, in the 
infinite size limit one finds an effective single spin equation, which in the special case of 
complete asymmetry will have no retarded self-interaction. This induces simplifications, 
resulting here in explicit self-consistent laws for order parameter functions. The further 
ansatz of time translation invariance leads to a tractable stationary state problem, from 
which, however, non-persistent order parameters cannot be fully removed. In addition 
to solving the full (exact) order parameter equations numerically, we also investigate 
an interpolative approximation that does depend on only a few scalar quantities and 
compare the predictions of both the full and the approximate theory with numerical 
simulation data. The interpolative theory is exact for zero a and also at zero T for 
small a (where small depends on the choice of gain function). We investigate the phase 
diagram of our model in the (a, T) plane, and examine the transition from non-recall to 
recall (we show that there is no spin-glass phase). Numerical simulations support our 
theoretical findings and predictions. 

2. Model definitions 

Our model describes a network of N continuous neurons Ui G 1R which evolve in time 
according the following Langevin equation 

-^Wi(t) = J ij9[Uj(t)} - Ui{t) + T]i(t), (1) 

where g[u] is an odd sigmoidal function, which saturates for u — > ±oo to ±1 
(respectively), and with the standard Gaussian white noise terms f]i(t) with moments 

(7fc(t)) = 0, (vMVjtt')) = 2TSijS(t - 1'). (2) 

The non-negative parameter T controls the level of noise, with T = and T = oo 
corresponding to deterministic and fully random dynamics, respectively. The process 
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(JTJ does not obey detailed balance (not even for symmetric J^-), and thus will never 
reach equilibrium. This rules out the techniques of equilibrium statistical mechanics, 
and we are forced to solve the dynamics. In the fashion of attractor neural networks we 
store p (randomly drawn) patterns £^ = . . . , £^) G { — 1, 1}^ in this system, with 
/U = 1, . . . ,p, by choosing diluted Hebbian-type interactions {Jij} as in [Zj: 

The (fixed) {%} define the connectivity, and are drawn at random according to 

% < j : P(cy) = ^8 Cijjl + (1 - ^)<^,Q (4) 

z > j : P(cy) = r8 Cij , Cji + (1 - r) + (1 - ^)5c !3 ,o] (5) 

The parameter c is seen to specify the average number of connections per spin, and 
r controls the symmetry of our architecture. We will choose extreme dilution, i.e. 
limjv^oo c/iV = \mi N _, 00 c~ 1 = 0, such as c = log(iV). The number of patterns p to be 
stored is expected to scale with c, so we define p = ac with a > finite. 

For simplicity, and in line with previous analyses, we will make the so-called 
'condensed ansatz' in solving the dynamics: we assume that the system state has an 
O(iV ) overlap only with a single pattern, say fi = I. This situation is induced by initial 
conditions: we take a randomly drawn u(0), generated by 



P(u(0)) = II j^ 1 + m M^(0) - tf) + \[l - m ]5( Ul (0) + C 1 )} 



(6) 



with m measuring the overlap between the initial state and the first pattern, m = 
N~ x ^2i(£,lui(0)) ■ The patterns /i > 1 and the architecture variables {%}, are viewed 
as disorder. We assume that for iV — > oo the macroscopic behaviour of the system is 
'self- averaging', i.e. only depends on the statistical properties of the disorder rather 
than its microscopic realization. Averages over the disorder will be written as 7TT . 



3. Generating functional analysis 

To analyze the dynamics following JE] we discretize time (in units of A), write the 
probability density of observing a microscopic 'path' {u(0), . . . ,u(t m )} through phase 
space as P[u(0), . . . , u(£ m )], and introduce the familiar generating function Za[V'] : 

Z A [il>] = (ex P {-iJ2Yl ^i(t)9[ui(t)}}) 

i<N t<t m 

= /du(0)...du(t w )P[u(0),...,u(t m )]exp{-iJ]5]AV'i(t)^K(t)]} (7) 
J i t 

Later we will put A — > Of. Note that (JJJ) differs from the standard definition in the 

appearance of the non-linear function g[u] in the exponent. For full details on the 

| The non-trivial technicalities related to the assumed commutation of the limits N — > oo and A — » 
are familiar issues in applying saddle-point arguments to path integrals, which we will not discuss here. 
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procedure for deriving from (J7J), in the limit iV — > oo, an effective single spin equation 
we refer to e.g. [T^]. The discretized version of (JIJ describes a Markov process, so 
the path probability density is a product of individual transition probability densities. 
In the ltd convention we do not pick up a Jacobian term from the discretization. We 
add time- dependent external fields 0i(t)£j to the deterministic forces in order to define 
response functions later, and proceed in the standard manner to a saddle point problem. 
Solution of the latter leads us to a closed macroscopic theory in terms of the familiar 
dynamic order parameters m(t) = linijv^oo iV -1 {g[ui(t)}) (the recall overlap), 
C(t,t') = liniTv^oo N^ 1 (g[ui(t)]g[uj(t')}) (the single-site correlation function) and 
G(t, t') = liniAr^oo ■h J2i 9g[ui(t)]/d^l9i(t) (the single-site response function). The order 
parameters are to be solved from the saddle-point equations 

m(t) = (g[u(t)]), (8) 
C(t,t') = (g[u(t)]g[u(t')]), (9) 
G(t,t') = d(g[u(t)])Jde(t') (10) 

Here the measure (. . is defined by the statistics of the following effective single spin 
process (having set i/> = and taken the limit A — > 0): 

d /"* 
u {t) = - u {t) + m{t) + 6{t) + ar &t' G{t,t')g[u{t')} + (j>{t) (11) 

J —oo 

where 4>(t) is a zero mean Gaussian process with covariance 

(0(t)0(t')) = 2T5(t - t') + aC(t, t'). (12) 

In the remainder of this paper we will consider r = only (i.e. fully asymmetric dilution, 
with Cjj independent of Cjj), so that the retarded self-interaction in (fTTj) vanishes. This 
also implies that there is no longer a need to solve for the response function, as the 
macroscopic laws ()8p9|) now close already for the order parameters {m(t),C(t,t')}. 

4. Solving the single spin equation for asymmetric dilution 

As our main interest will be in stationary state properties, we choose initial conditions 
at to = — oo. For r = we can readily integrate the stochastic equation (fTTj) . giving 

u (t) = k{t) + Z(t) (13) 
k(t) = f dt' e-^[m(t')+e(t')} Z(t) = [ dt' e-^'Uit') (14) 



So Z(t) is also a zero mean Gaussian noise, now with covariance (Z(t)Z(t')) = E(t,t') 
E(t, t') = Te- 1 '-*' 1 + a f ds e^- s) f ds' e~ {t '"^(s, s') (15) 

J —oo J —oo 

We may use (|13I14|) and the standard abbreviation Dz = (2ix)~ l l 2 e~^ z2 dz for Gaussian 
measures to write the closed equations ()8I9|) in the following explicit form: 

m(t) = J Dx g[k(t) + xy/Z(t,t)] (16) 
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C(t, t') = J DxDy g[k(t) + ai(t, t')(a 2 (t, t')x + a 3 (t, t')y)} 
x g[k(t) + a x (t,t'){a 2 {t,t')x + a^t^y)] 
in which (using the symmetry of S): 



(17) 



ai (t,t') = l/^E(t,t)+E(t',t')-2E(t,t') 

a 2 (t,t') = y/~(t,t)~(t',t')-~(t,t')2 
a 3 {t,t') = E{t,t)-E{t,t') 
a 4 (t,t') = E(t',t)-E(t',t') 

Let us turn to time translation invariant solutions (TTI), where m(t) = m, 0(t) = 9 
and C(t,t') = C(t — t'), and consequently E(t,t') = E(t — t'). This will simplify our 
analysis, and experience with asymmetric systems suggests that most, if not all, solutions 
will asymptotically obey TTI. We now find that equations (|16I17J) simplify to 



m 



C{t) 



J Dx g[m + 9 + xy/E(0)) 



(18) 



DxDy g 



X g 



m 



+ 9 + xJME(0) + S(r)] + yJmo) ~ S(r)] 



T 



ra + 9 + o;^i|H(0) + H(r)] - y^\[E{Q) - E{r)\ 
Te" |T| + -a / du e - |u+r| C(u) 

2 J-oo 



(19) 
(20) 



We note that, upon repeated differentiation with respect to r, one can also derive from 
this latter equation the following identity: 

E"(r) = S(t) - aC(r) (21) 

In detailed balance models at this stage one would be able to transform away the non- 
persistent order parameters, and derive closed equations for the persistent objects m, 
q = C(0) and q = C(oo). Here such a reduction is not possible. To appreciate this we 
first rewrite ()19)1 (by suitably transformations of integration variables) as 

in + 9 + x^E{j) + yy/E(0) - E(r) 



C{r) 



Dx 



Dy g 



(22) 



Using the relation E(oo) = aq and the short-hand H(0) = k, our expressions for the 
persistent quantities {m, go, <?} then take the form 



with 



m — J Dx g[m + 9 + x^/k\, q = J Dx g 2 [m + 9 + x\[k\ 
q = J Dx | J Dy g [m + 9 + x^faq + y\J k — aqj | 

POO 

K = T + a dr e" r C(r) 
Jo 



(23) 
(24) 

(25) 
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Unless T = (where one has solutions with q = q Q and C{r) = q for all r, to which we 
will return later), the term k can not be expressed in terms of {m, qo,q}, but requires 
knowledge of non-persistent correlations C(r). From this stage onwards we will take 
9 = 0, in order to study different solution types and transitions between them. 



5. Special solutions 

5.1. Non-recall TTI solutions 

Non-recall solutions are those where m = (to be expected for large T), and where in 
the TTI regime we are left only with a closed set of equations for C(r): 



C(r) 



DxDy g 



x 9 



x 



|[H(0)+H(r)]+y,/i[H(0)-H(r)] 



S(r) = Te~ |T| + -a / rfu e - |u+T| C(n) 



(26) 
(27) 



The simplest case is T — > oo. Since C(r) is bounded, we may expand our equations and 
find that the second term in (|2*7j) is negligible relative to the first. This results in 



lim C(r) 

1 — >oo 



arcsm e 



7T 



(2? 



0. It is a trivial matter to show that 



Clearly lim^oo C(0) = 1 and limr-*oo C{oo] 
this is the only possible solution for T — » oo. 

In the case where g[u] = sgn[w] we can push the analysis of (jz 



further for 

arbitrary T, as we may do the integrations in (|2*6^) explicitly (in the recall phase this 
would not have been possible). For g[u] = sgn [it] we may write 



C(t) = J DxDy sgn[x + tan(^)i/]sgn[x — taii(ip)y] (29) 
tan 2 W = [H(0)-H(r)]/[H(0) + H(r)] (30) 
Upon writing (x, y) in polar coordinates, the integrations can be done, and we find 



sin(-7rC(r)) 



Te"M + f J^du e-\ u+T \C{ 



u 



T + zf^du e-MC(«) 
For T — ► oo we recover ()28jl . Putting r — > oo in (j!TT| gives 



sin(-7rC(oo)) 



aC(oo) 



T + af^du e~ u C( 



u 



(31) 



(32) 



The paramagnetic (P) state C(oo) = always solves (J3XJ) , as expected. It also follows 
from (|31|) that a continuous transition from a paramagnetic to a spin-glass (SG) state 
{m = 0, C(oo) > 0} would occur (unless preceded by a recall transition) at k c = 2a /n. 
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5.2. Detailed balance type solutions at zero temperature 

Let inspect whether and where we have detailed balance type macroscopic laws, where 
after elimination of transient parts one finds 'effective' equations with C(r > 0) = q. 
Here the ansatz C(r > 0) = q implies that H(r) = Te~' T ' + aq and n = T + aq. One 
now finds that it gives proper solutions of our order parameter equations if and only if 
T = 0, in which case we obtain (for 6 = 0) q = q and the familiar looking 



For g[u] = tanh[/3w] these equations (jHHjl are identical to those of the symmetrically 
extremely diluted attractor network with binary neurons [§]. For g[u] = sgn[w] we 
immediately find the three possible solutions P {m — 0, q — 0}, SG {m — 0, q — 1} 
(both P and SG exist for any a) and R {m = m*, q = 1}, where m* is the solution of 

m = Erf [m/V2a] (34) 

The retrieval solution bifurcates from the SG one at a c = 2/n « 0.637. 

The constant correlation function implies that the spins are microscopically frozen, 
thus there are no probability currents at all in the system so detailed balance is trivially 
restored. It is clear that such solutions always exist and are stable at a = 0. 

We next inspect the local stability of the above special T = solutions against 
perturbations away from C(t > 0) = q for a > 0. We thus put m — >• m + 8m and 
H(t) — > q + 52 (r) (with Sm and 5H(t) small) and find the eigenvalue problem 

<JS(r) = / du 6E(u)[- — ^-e" 1 " 1 + e -l— -I] (35) 



oc 



1-Ko 



K x = \a iDx \g'[y}\ (36) 



where, with y(x) = m + xJolq 



2 



zfDx g'[y}.fDxg[y}g"[y] 



K 2 = a Dx g[y]g"[y} + * (37) 

y l-JDxg'[y\ 

The only eigenfuction of (|35p which is not trivial and bounded for r — > 00 is given by 

AT, 



5S(r) = cos(v / 2ir 1 - lr) - j _ — 2 _ — (38) 

with K\ > |. Since lim^o ^1 = 0, the bifurcation condition becomes ffi = |, or 

1 = a |^[m + x^aq^Y (39) 

The size of the region along the T = axis where the solution (|33J) is stable is dependent 
on our choice of the gain function g. For g[u] = sgn[u] we find that there is never a 
continuous transition while for g[u] = tanhjju] there can be and the solution (}3*3*j) may 
be stable in a large part of the recall region. To see more clearly why there are no 
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continuous transitions for g[u] = sgn[w] we note that in that case the special solution is 
Q — Qo — 1; an d our eigenvalue problem becomes 

a + SS(t) = f [°° in e-'-l / Dx Erf* ( m ±fi§ + |p) (40) 

For small perturbations the argument of the error function will be large and we can use 
the expansion (22] 

Erf (*) = 1 " ~!- e ~ x2 lt( 1 -T-2+ 77r4 " T^Jrs +■■) ( 41 ) 
v ' y/rrx 2x (2x 2 ) 2 (2x 2 ) 3 / v ' 

to see that on the right-hand side of (|40|) there is no polynomial term in the perturbation, 
and hence no continuous bifurcation. 



6. Non-equilibrium phase transitions 

6.1. Continuous transitions without anomalous response 

Continuous transitions from P or SG states to recall states (R, where m > 0) or from P 
states to SG states, are found upon expanding either (J23j) or (|24*j) (both with = 0) for 
small m or small q, respectively. This is found to give the simple bifurcation conditions 

P -> R, SG -> R : y^Dx ; [aV«| = 1 (42) 

P -> SG : ^[ar-s/w] = VV" (43) 

For = sgn[w] these conditions reduce to k c = 2/n and k c = 2a /ir, respectively. 
Here, since k G [T, T + a%], we are sure that continuous transitions P-^R or SG— >R 
can occur at most for T G [2/tt — a, 2/7r], and that continuous transitions P-^SG can 
occur only when T < 2a /it. To determine which instability away from the paramagnetic 
state occurs first we need to inspect the dependence of k, as defined in (|23j) on (a, T). 



6.2. Transitions to states with anomalous response 

We next inspect the solution of our full TTI equations for large times, upon separating 
non-persistent from persistent terms in the functions H(r) from C(t): 

C(r)=q + C(r), E(r) = aq + E(r) (44) 

Insertion into our equations for 5(r) and C(r) gives 



1 f°° 

S(r) = Te" |r| + -ay rfw e - |u+T| C( 
C{r) = Idx{ [Dyg 



m + xy aq + H(r) + W k - aq - 5(r) 



- 9 



(45) 
(46) 
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For large times the non-persistent terms will be small, so we may expand the right-hand 
side of (|4*U|) in powers of H(r). First we work out the nonlinearity g[. . .]: 

g[. . .] — g \m + x^faq + y\/ k — acq] 



X 



y 



g' \m + xJaq + y^n - aq] + 0( 

laq ^/n — aq] J 

which gives, after partial integration over the Gaussian disorder variables, 
C(t) 



A 



A ~(r) + 0(~ 2 (r)) 
jDx{jDyg [m + x^q + y^^]) 



(47) 



(48) 



Dy 



g [m + Xyfaq + 'k — aq\ | 



/aq y/n — aq 

= jD x {Jny g -[ m + x ^- q + yV^\} (49) 
For sufficiently large times we may therefore replace the duo ()45|46|) by the leading order 
C(t) = A |Te" |T| + ^a J du e" 1 ^ 1 ^^) j + . . . (50) 

This equation is solved asymptotically by C(r) ~ e _7T , with exponent 7 = y/l — aA. 
We must conclude that a transition to a regime with anomalous response (which in 
equilibrium disordered systems would correspond to the AT line |2I]) occurs when 
aA = 1, i.e. when 

(i / U.i' < / JJy <f \m -\- - (\<i + 1; - <\<i\ } ■ 1 (51) 



J Dx I J Dy g' [m + x^faq + y\J ' k — aq] | = 1 



We see that in the paramagnetic state m = q = the condition (Jo"Tj) coincides with 
that of the P— >SG transition (|4*H|) . and that for T = it coincides with the instability 
condition (JH9"j) . Again, for the special choice g[u] = sgn[w] all integrals can be done 
analytically, and we find ()51)1 taking the more explicit form 



ae 



-m 2 /(K,+aq) _ \^ 

2 v 



a 2 q 2 



(52) 



7. Bounds and approximations for the stationary state 



7.1. Arbitrary non-linearities 

We established that it is not possible to obtain closed equations for persistent order 
parameters only, due to the dependence of (f25|) on the short-time part of C(r). However, 
we know that C{r) decays from C(0) = q to C(oo) = q, so that (|23|) can be written as 

k = T + aq + aA( y q -q), A E [0, 1] (53) 

with A w 1 when the decay of C(r) is much slower than exp[— r], and A smaller when 
the decay is fast. Thus we know that our phase diagram must interpolate between those 
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obtained for the extreme cases A £ {0, 1}, with these cases becoming identical both for 
T — > at small a (where go — g ^ before the transition to non-flat C(r)) and a — > 0: 



'Slow' limit, A = 1 : m 



'Fast' limit, A = 



J Dx g[m + xy/T + aq ] (54) 

y g 2 [m + x\/T + ago] (55) 

| y g[m + xV^? + y-y/T + a(q — q)] | (56) 

jDxrtn + xy/T + ZSl ^ (58) 

g = y Dx |y g[m + Xy/aq + yVT]^ (59) 



9o 

m 
qo 



Rather than analyze the extreme bounding cases A £ {0, 1}, one could construct a 
rational interpolation between A = and A = 1 by using our knowledge of the long- 
time behaviour of the non-persistent correlations as established in the derivation of 
(I51|) . viz. C[t) ~ g-rVi-aA w ith A as given in ()49j). to define the simplest function 
which satisfies both the long-time profile and the initial conditions C(0) = qo — q: 
C{t) = (qo — q)e~ T ^ 1 ~ aA . This implies the following approximate expression for A: 

1 



dr e -[Wra] = _ 

o 1 + VI - "A 



Equivalently: 



n = T + aq + 



a (g - g) 



1 + VI - "A 
This then leads us to the following theory: 



interpolation : m 



Qo 



A 



Dx g 
Dx g 2 
Dx 
Dx 



m + x\ \T + aq + 



a (g - g) 



1 + VI - "A 



m + x\ \T + ag + 



a (go - g) 



1 + VI - «AJ 



m + xV«g + y\ T ■ 



a (g - g) 



1 + Vl - «A 



m + xJTxq +y\ IT + 



a (go - g) 



(60) 
(61) 

(62) 
(63) 
(64) 
(65) 



1 + VI - aA 

Again we note that these equations are exact in either of the limits T —>■ and a — > 0. 
7.1?. Predictions for g[u] = sgn[w] 

For the choice g[u] = sgn[w], where Gaussian integrals can be done, we have go = 1. Thus 
the slow limit gives the P — > F transition line k = a + T = 2/tt. In the paramagnetic 
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Figure 1. Phase diagram for the asymmetrically dilute graded response model. The 
dashed line is the transition predicted by the interpolation approximation while the 
dotted line is that predicted by numerically solving for the correlation function. The 
interpolativc method gives reasonable agreement but systematically underestimates 
the size of the recall region with the error increasing as a function of a. 



state q = thus the fast limit gives a P— > F transition for a<latT = 2/7r and a P 
— > SG transition for a > 1 at T = 2a /it. In addition, the above set of interpolating 
equations can be simplified to 




Q 



Dx Erf 



2 m + x^faq 



a 



y/2(K - aq) 



{k, 2 - a 2 q 2 )\ + {^k 2 - a 2 q 2 - ^ e - m2 /(«+^))l 
with the phase transitions occurring at: 

2 



P -> R, SG -> R : 
P -> SG : 



TV 

2a 

7T 



Anomalous Response : \/ k 2 — a 2 q 2 



^ a c -m 2 /(K+ag) 
71 



(66) 
(67) 

(68) 
(69) 
(70) 



In the paramagnetic state m = q = 0, where the conditions ()69I70|) become identical, 
equation (jfi7j) is seen to simplify significantly, and can in fact be solved for k: 



T{l-l) + \a+^T 2 + aT{l-l) + \a 2 
2(1 -h) 



(71) 



Given this expression we may in turn derive explicit expressions for the our phase 
transition transition lines. One finds that the transition P— >SG cannot occur, and that 
the P— >R transition line reduces to 

2 



R 



T c (a) = y/1 — a — 1 



7Y 



(72) 
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This critical temperature decreases with increasing a from its maximum value T c (0) = 
2/n « 0.637 down to T c (a c ) = 0, with a c = ^(1 — -) ~ 0.868. Since our interpolation 
equations are exact at T = and at a = 0, we can be sure that also the two critical 
values T c (0) = 2/n and a c are exact. 

Thus, given that one always enters a recall phase directly (without a spin-glass 
phase, unlike symmetrically connected models), the only remaining transition to be 
investigated is the one marking the possible onset of anomalous response inside the 
retrieval phase. Combining ()67j) with ()7()j1 shows that at this transition k = T + a, so 
that we have to solve the transition line from the following trio of coupled equations 

r m i 

Anomalous Response : m = Erf — = (73) 

V2(T + a)J 



Dx Erf 2 



m + xJ~aq 



y/2[T + a(l-q)] 



(74) 



4a 2 2m 2 

[T + a(l - g)] [T + a(l + g)] = — =-e r +<*d+*) (75) 

7T Z 



Examining (J25J1 we see that for an anomolous response we require q — 1 which via 
(j74|75|) implies T = a = is the only possibility. However, there one trivially has 
C(t) = 1, W and there is no transient term in the correlation function. 

We plot the phase diagram Figure [TJ both using the approximation (|72|) and by 
solving for the whole correlation function numerically. We see that the results are 
similar although not identical, the interpolation method seems to predicts a slightly 
smaller recall region. 



8. Comparison with simulations 

In Figures El and El we examine both the distribution of spins at a single time and the 
correlation function. In simulations the spins evolved for O(10 5 ) discretized time steps 
(with At = 0.02) in order that they reached their static distribution. It is clear from 
(|13|) that the static distribution of spins must be Gaussian with mean m and variance 
k. Our two theories (exact versus interpolation) give different values for m and k, yet 
both are seen to agree very well with the simulations as far as the spin distribution is 
concerned. For the correlation function, however, it appears that solving the full order 
parameter equations numerically gives excellent agreement with simulations and is a 
significantly better guide to both the persistent correlation and the short time shape 
than interpolation. The differences between the two theories appear to increase with a 
within the recall regime. 

In Figure 0] we plot magnetisations m and persistent correlations q for a variety of 
values for T and a on both sides of the predicted phase boundary, for both simulations 
and theory. In the left picture we see that for moderate a(= 0.2) both theories give 
good agreement with each other and with simulations. However, in the right picture, 
i.e. at higher a near the phase transition, again the numerical solution of the full order 
parameter equations gives better agreement. 
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Figure 2. The left graph depicts the static distribution of spins at a = T = 0.25. 
The solid line is from a simulation of TV = 64000 spins, the dotted line is from solving 
for the correlation function and the dashed line is from the interpolation theory. Both 
theories give excellent agreement with the simulation. The right graph depicts the 
corresponding TTI correlation function as a function of the number of discretized time 
steps t/At (with here At = 0.02), with the meaning of the different lines are as in the 
left picture. It is clear that the solving for the correlation function numerically gives a 
significantly better calculation of the correlation function. 




-10 -5 5 10 250 500 750 1000 

u t/At 

Figure 3. The left graph depicts the static distribution of spins at a = 0.5 and 
T = 0.125. The solid line is from a simulation of N = 64000 spins, the dotted line is 
from solving for the correlation function and the dashed line is from the interpolation 
theory. Both theories give excellent agreement with the simulation. The right graph 
depicts the corresponding TTI correlation function as a function of the number of 
discretized time steps t/At (with here At = 0.02), with the meaning of the different 
lines are as in the left picture. It is clear that the solving for the correlation function 
numerically gives a significantly better calculation of the correlation function. 
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Figure 4. The left graph shows the magnetisation versus temperature for a = 0.2. 
The solid line with o markers is averaged over 5 simulations of N — 64000 spins 
(error bars are all < 0.01), the dotted line with □ markers is obtained via solving the 
correlation function numerically, and the dashed line with x markers is calculated from 
the interpolation theory. The vertical lines mark the predicted phase transitions, the 
left line being from the interpolation. Both theories give excellent agreement with the 
simulation. The right graph shows the persistent correlation versus alpha at T — 0.25. 
The different lines are as above. Solving the correlation function numerically gives 
better agreement with the simulations at higher a. 

9. Discussion 

In this paper we have used the formalism of generating functional theory to solve 
the dynamics of an extremely dilute asymmetrically structured Hopfield-type attractor 
neural network model with Langevin dynamics near saturation. This is a non-detailed- 
balance problem. In contrast to extremely diluted models with synchronous dynamics, 
we here cannot benefit from the Gaussian fields since we need more than just the 
marginal field distribution. We have looked for time translationally invariant solutions 
as an aid to finding stationary solutions to this problem. In general we find that we 
need a full functional order parameter C (r) to describe this stationary solution exactly. 
We have also looked at an interpolative theory that, while only exact at a — 0, depends 
on only a few scalar parameters rather than an entire order parameter function. This 
reduced theory gives good agreement at low a but shows discrepancies at higher values 
of a. 

As expected for this type of architecture the phase diagrams exhibit just two phases, 
a recall phase and a non-recall phase. There are no spin glass phases, which is due 
to the lack of a retarded self-interaction in the effective single spin equation. We 
have calculated, for specific choices of the gain function g, the system's (a,T) phase 
diagram, and have found the second order transitions in this diagram via a bifurcation 
analysis. The methods used in this paper to solve the effective single spin equation 
could be valid for other problems with a similar architecture and similar dynamics. The 
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network's connection asymmetry is obviously an assumption whose justification is model 
dependent, here it is reasonable and leads to exactly solvable statics. 

It would also be of interest to study the initial dynamics of this problem, as that 
will be of more relevance to biological experiments. Other generalizations could include 
moving to a finitely connected architecture which may be closer to that seen in nature. 
Here the order parameters would become single spin path probabilities, and the effect on 
these path probabilities of a change in the external field at each point on the path [24J. 
It may also be of interest to see if an approximate theory could be constructed when 
arbitrary degrees of symmetry are introduced into the problem, such that the method 
presently used in this paper is no longer accessible. 
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